This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

If we want to go full daemonless, containerd daemon dependency can eventually be removed, it just keeps the scope of this PR smaller for this iteration.

openshell-driver-native

Shouldn't it be named openshell-driver-containerd ?

This driver — not containerd — spawns the low-level runtime. containerd is used only for image pull/unpack and snapshot management (Transfer/Images/Content/Snapshots services); it never creates a containerd Container or Task object, and its io.containerd.runc.v2 shim is never involved. The driver builds a standard OCI bundle (config.json + a mounted rootfs/) from the snapshot containerd prepared and drives the configured runtime directly through its own create/start/state/kill/delete CLI contract — the same integration pattern containerd's shim,

Oh. That seems like it needs a really strong justification.

This relates to #981 - I still would really like to move away from openshell driving lots of low level custom sandboxing itself towards one where it just reuses existing container framework tooling - especially where e.g. the proxy is a sidecar or other associated process etc.

openshell-driver-native

Shouldn't it be named openshell-driver-containerd ?

This driver — not containerd — spawns the low-level runtime. containerd is used only for image pull/unpack and snapshot management (Transfer/Images/Content/Snapshots services); it never creates a containerd Container or Task object, and its io.containerd.runc.v2 shim is never involved. The driver builds a standard OCI bundle (config.json + a mounted rootfs/) from the snapshot containerd prepared and drives the configured runtime directly through its own create/start/state/kill/delete CLI contract — the same integration pattern containerd's shim,

Oh. That seems like it needs a really strong justification.

This relates to #981 - I still would really like to move away from openshell driving lots of low level custom sandboxing itself towards one where it just reuses existing container framework tooling - especially where e.g. the proxy is a sidecar or other associated process etc.

containerd-daemon dependency might eventually go away, it's undecided, it just keeps the implementation smaller for now, it only handles transport, runc or crun is orchestrated directly by OpenShell.

This is sorta linked to that:

#2313

oci-client crate in the Rust ecosystem I think, is not the right call.

docker, podman, k8s remain separate drivers.

But thanks @cgwalters I did rename:

feat(driver-native): add containerd-backed native compute driver

to

feat(driver-native): add native compute driver

the "containerd" part of the name does distract from the purpose here. It's OpenShell that kicks off the runc's or crun's. containerd has nothing to do with the isolation.

@ericcurtin

oci-client crate in the Rust ecosystem I think, is not the right call.

Would you expand on this a bit?

@ericcurtin

oci-client crate in the Rust ecosystem I think, is not the right call.

Would you expand on this a bit?

@krishicks see here, it's described in more detail:

#2313

@ericcurtin

oci-client crate in the Rust ecosystem I think, is not the right call.

Would you expand on this a bit?

@krishicks see here, it's described in more detail:

#2313

It's not really explained there either:

Using oci-client crate is a mistake.

What's the problem that's being solved in #2313, and why is oci-client a mistake or not the right call?

@ericcurtin

oci-client crate in the Rust ecosystem I think, is not the right call.

Would you expand on this a bit?

@krishicks see here, it's described in more detail:
#2313

It's not really explained there either:

Using oci-client crate is a mistake.

What's the problem that's being solved in #2313, and why is oci-client a mistake or not the right call?

The OCI protocol is ossified... This new Rust crate is used in projects like oras, which is known to be unreliable compared to implementations based on containerd or containers/image

It can be reliable on OCI registries like Docker Hub, but it's the internal ones where you get the bug reports, hand-rolled OCI clients reliably break on obscure auth schemes, manifest quirks, and layer edge cases

I'm a bit skeptical of the oci-client crate being problematic as far as talking to registries, I think the larger picture thing is ecosystem stuff like finding and using pull secrets, which is outside the scope of that crate. And container storage.

Thanks for putting this together. As discussed in our sync, the goal of an
OCI-based driver is to provide a smooth out-of-the-box, single-node
experience where the VM driver is unavailable -- for example, when nested
virtualisation is not supported.

Using containerd for reliable registry access, content storage, unpacking, and
snapshots seems reasonable while we assess feasibility. I do not, however,
consider a required system containerd daemon to be a viable long-term design.
It introduces another installed and running service—including its configuration,
socket, namespaces, snapshotter, leases, compatibility, and operational
lifecycle—solely to construct root filesystems before OpenShell invokes the OCI
runtime itself. Image management does not inherently require the containerd
daemon: the relevant registry, content, unpacking, and archive functionality can
be provided through containerd libraries, as #2313 explores, or through another
OCI image implementation.

That makes it especially important for the current implementation to make
containerd's role explicit and temporary. Containerd is not responsible for
sandbox process lifecycle or provisioning here: OpenShell constructs the OCI
bundle and directly manages runc/crun, while containerd acts as one rootfs
provider. Its API and lifecycle concepts should remain behind that provider
boundary rather than becoming part of the compute-driver contract. Structuring
the code this way would make the design easier to understand and leave room for
replacing the daemon-backed provider without redesigning the OCI provisioner.

Conceptually, I would like to see something along these lines:

Local Compute Driver
├── RootfsProvider
│   ├── ContainerdRootfsProvider
│   ├── VmRootfsProvider
│   └── Future daemonless OCI provider
└── SandboxProvisioner
    ├── OCI/runc
    └── VM/libkrun

Each provider may produce a backend-specific artifact—for example, OCI mounts
for runc or an ext4 block image for libkrun — while sharing the preparation and
lifecycle contract.

I also think we should explicitly evaluate how much of the existing VM rootfs
pipeline can be adapted for the OCI provisioner. The final VM artifact is an
ext4 block image and is not directly useful to runc, but the earlier stages—
image resolution, acquisition, verification, unpacking, cache identity, and
progress reporting—are not inherently VM-specific. A useful internal split may
therefore be an ImageSource, which produces a verified OCI layout or unpacked
rootfs, followed by a backend-specific RootfsMaterializer, which produces
either OCI mounts or a VM disk image.

There are also parts of the VM driver that we should consider extracting instead
of implementing them again for OCI:

• Collision-aware subnet allocation
• Host networking and nftables lifecycle
• Translation of the configured gateway endpoint into an address visible from the sandbox
• Supervisor environment and credential construction
• Transactional allocation and cleanup of rootfs, network, and runtime resources

The VM-specific ext4, boot, libkrun, and VFIO code would remain in the VM
provisioner. Likewise, OCI specs, namespaces, capabilities, cgroups, and runc
state management would remain in the OCI provisioner. In both cases, the
provisioner creates the outer sandbox environment and starts the supervisor;
the supervisor then applies the RFC-defined isolation backend to the untrusted
agent workload.

While reviewing the implementation, I also found several issues that appear to
block the current version:

• The default supervisor endpoint is 127.0.0.1, but the supervisor runs in a separate network namespace. It will therefore connect to its own loopback rather than the gateway.
• Sandbox names are used directly as path components. The standalone gRPC service only checks that they are nonempty, which permits path traversal into operations such as remove_dir_all.
• Errors after mounting the rootfs can leave the mount and bundle behind. Snapshot cleanup can then fail because the snapshot remains mounted, and subsequent creation reports AlreadyExists.
• Network allocation hashes sandboxes into only 64 slots without collision detection. Different sandboxes can receive identical IP addresses and routes.
• Direct runtime invocations use the default global runc state root instead of an OpenShell-specific runtime root.
• Sandbox token files are not created with the restricted directory and file permissions used by the existing drivers.

The unit tests cover the individual helpers well, but the real containerd/runtime tests are ignored and there is no native gateway end-to-end lane. I think an end-to-end test using the real supervisor is important before merging because it would catch the endpoint and lifecycle issues above.

My preference would be to treat this PR as the start of an OCI sandbox
provisioner and a temporary containerd rootfs provider, rather than introducing
another monolithic compute driver or making the containerd daemon a lasting
architectural dependency. That would clarify containerd's limited role, allow
useful VM-driver infrastructure to be shared, and better align the
implementation with the architecture we discussed and the terminology in RFC
0001.

Given the architectural questions and correctness issues above, I do not think
the PR should merge in its current form. I would prefer that we first establish
these boundaries either by restructuring this PR or agreeing on smaller
staged changes.

Thanks for putting this together. As discussed in our sync, the goal of an OCI-based driver is to provide a smooth out-of-the-box, single-node experience where the VM driver is unavailable -- for example, when nested virtualisation is not supported.

Using containerd for reliable registry access, content storage, unpacking, and snapshots seems reasonable while we assess feasibility. I do not, however, consider a required system containerd daemon to be a viable long-term design. It introduces another installed and running service—including its configuration, socket, namespaces, snapshotter, leases, compatibility, and operational lifecycle—solely to construct root filesystems before OpenShell invokes the OCI runtime itself. Image management does not inherently require the containerd daemon: the relevant registry, content, unpacking, and archive functionality can be provided through containerd libraries, as #2313 explores, or through another OCI image implementation.

"I do not, however, consider a required system containerd daemon to be a viable long-term design.". I agree here.

I've been waiting for opinions and decisions on:

#2313

before changing that here. I might not have communicated that great here. The three daemonless, out-of-the-box, single-node options, I can think of are:

1. Use containerd as a libloading lib
2. Use https://github.com/podman-container-tools/container-libs as libloading lib/libs sort of what @cgwalters is suggesting, but I think he may be more leaning towards, just use docker/podman in this case, we have an alternate driver for that fatter option already though. I'm not sure this hits the goals of people like @jmabry who's been looking at bubblewrap and apptainer though. Also, containerd has more functionality multi-platform accross macOS/windows/linux. I've used it multiple times for multi-platform transport.
3. Reimplement everything in Rust.

I think 3. carries the most risk FWIW. And would likely end up in a multi-year effort.

That makes it especially important for the current implementation to make containerd's role explicit and temporary. Containerd is not responsible for sandbox process lifecycle or provisioning here: OpenShell constructs the OCI bundle and directly manages runc/crun, while containerd acts as one rootfs provider. Its API and lifecycle concepts should remain behind that provider boundary rather than becoming part of the compute-driver contract. Structuring the code this way would make the design easier to understand and leave room for replacing the daemon-backed provider without redesigning the OCI provisioner.

Conceptually, I would like to see something along these lines:

Local Compute Driver
├── RootfsProvider
│   ├── ContainerdRootfsProvider
│   ├── VmRootfsProvider
│   └── Future daemonless OCI provider
└── SandboxProvisioner
    ├── OCI/runc
    └── VM/libkrun

Each provider may produce a backend-specific artifact—for example, OCI mounts for runc or an ext4 block image for libkrun — while sharing the preparation and lifecycle contract.

I also think we should explicitly evaluate how much of the existing VM rootfs pipeline can be adapted for the OCI provisioner. The final VM artifact is an ext4 block image and is not directly useful to runc, but the earlier stages— image resolution, acquisition, verification, unpacking, cache identity, and progress reporting—are not inherently VM-specific. A useful internal split may therefore be an ImageSource, which produces a verified OCI layout or unpacked rootfs, followed by a backend-specific RootfsMaterializer, which produces either OCI mounts or a VM disk image.

There are also parts of the VM driver that we should consider extracting instead of implementing them again for OCI:

• Collision-aware subnet allocation
• Host networking and nftables lifecycle
• Translation of the configured gateway endpoint into an address visible from the sandbox
• Supervisor environment and credential construction
• Transactional allocation and cleanup of rootfs, network, and runtime resources

The VM-specific ext4, boot, libkrun, and VFIO code would remain in the VM provisioner. Likewise, OCI specs, namespaces, capabilities, cgroups, and runc state management would remain in the OCI provisioner. In both cases, the provisioner creates the outer sandbox environment and starts the supervisor; the supervisor then applies the RFC-defined isolation backend to the untrusted agent workload.

While reviewing the implementation, I also found several issues that appear to block the current version:

• The default supervisor endpoint is 127.0.0.1, but the supervisor runs in a separate network namespace. It will therefore connect to its own loopback rather than the gateway.
• Sandbox names are used directly as path components. The standalone gRPC service only checks that they are nonempty, which permits path traversal into operations such as remove_dir_all.
• Errors after mounting the rootfs can leave the mount and bundle behind. Snapshot cleanup can then fail because the snapshot remains mounted, and subsequent creation reports AlreadyExists.
• Network allocation hashes sandboxes into only 64 slots without collision detection. Different sandboxes can receive identical IP addresses and routes.
• Direct runtime invocations use the default global runc state root instead of an OpenShell-specific runtime root.
• Sandbox token files are not created with the restricted directory and file permissions used by the existing drivers.

The unit tests cover the individual helpers well, but the real containerd/runtime tests are ignored and there is no native gateway end-to-end lane. I think an end-to-end test using the real supervisor is important before merging because it would catch the endpoint and lifecycle issues above.

My preference would be to treat this PR as the start of an OCI sandbox provisioner and a temporary containerd rootfs provider, rather than introducing another monolithic compute driver or making the containerd daemon a lasting architectural dependency. That would clarify containerd's limited role, allow useful VM-driver infrastructure to be shared, and better align the implementation with the architecture we discussed and the terminology in RFC 0001.

Given the architectural questions and correctness issues above, I do not think the PR should merge in its current form. I would prefer that we first establish these boundaries either by restructuring this PR or agreeing on smaller staged changes.

The rest sounds ok, I think we can figure it out.

@elezar pushed a round of fixes for the correctness issues you flagged (loopback endpoint, path traversal, runtime --root, subnet collisions, cleanup-on-error, token permissions), all with new unit tests. Left the RootfsProvider/SandboxProvisioner restructuring out of this push since it's a bigger design conversation. Want to spin that off as a follow-up issue/RFC before I touch the driver boundary?

before changing that here. I might not have communicated that great here. The three daemonless, out-of-the-box, single-node options, I can think of are:

Is reusing / adapting the implementation the VM uses an option? While I understand that #2313 raises questions about the robustness of the implementation and the long-term maintenance costs, it may be useful to separate the discussion around what we use for rootfs construction (home-rolled vs containerd lib) from what we're using to provision the sandboxes (libkrun vs OCI).

before changing that here. I might not have communicated that great here. The three daemonless, out-of-the-box, single-node options, I can think of are:

Is reusing / adapting the implementation the VM uses an option? While I understand that #2313 raises questions about the robustness of the implementation and the long-term maintenance costs, it may be useful to separate the discussion around what we use for rootfs construction (home-rolled vs containerd lib) from what we're using to provision the sandboxes (libkrun vs OCI).

Yes I agree, that's an option and should be the end goal. Consolidate on whatever the transport is.